Automated Program Repair Techniques Research Articles

A method to identify overfitting program repair patches based on expression tree

The primary aim of Automatic Program Repair (APR) is to automatically repair defective programs, with the intention of reducing the amount of effort required by developers. However, APR techniques may produce overfitting patches that do not truly repair the program, allowing the program to pass all test cases. This paper provides a comprehensive review of the overfitting problem and adds to the existing research on overfitting in conditional statements. Our proposed method, ETPAT (Expression Tree-based Patch Assessment Technique), implements expression trees and targeted coverage criteria to identify differences between the original and the patched program. We utilize ETPAT to verify test case adequacy. In parallel, ETPAT also guides the generation of corresponding test cases via equivalence class information, which may be added to the original test suite, making it more robust while also preventing the repair technique from generating comparable overfitting patches. With reference to the patch set in the BuggyJavaJML benchmark, ETPAT recognized 77/82 (93.9%) overfitting patches out of 120 patches related to conditional constraints, displaying superior accuracy rates and fewer test cases required than the original repair tool.

A Survey of Learning-based Automated Program Repair

Automated program repair (APR) aims to fix software bugs automatically and plays a crucial role in software development and maintenance. With the recent advances in deep learning (DL), an increasing number of APR techniques have been proposed to leverage neural networks to learn bug-fixing patterns from massive open-source code repositories. Such learning-based techniques usually treat APR as a neural machine translation (NMT) task, where buggy code snippets (i.e., source language) are translated into fixed code snippets (i.e., target language) automatically. Benefiting from the powerful capability of DL to learn hidden relationships from previous bug-fixing datasets, learning-based APR techniques have achieved remarkable performance. In this article, we provide a systematic survey to summarize the current state-of-the-art research in the learning-based APR community. We illustrate the general workflow of learning-based APR techniques and detail the crucial components, including fault localization, patch generation, patch ranking, patch validation, and patch correctness phases. We then discuss the widely adopted datasets and evaluation metrics and outline existing empirical studies. We discuss several critical aspects of learning-based APR techniques, such as repair domains, industrial deployment, and the open science issue. We highlight several practical guidelines on applying DL techniques for future APR studies, such as exploring explainable patch generation and utilizing code features. Overall, our article can help researchers gain a comprehensive understanding about the achievements of the existing learning-based APR techniques and promote the practical application of these techniques. Our artifacts are publicly available at the repository: https://github.com/iSEngLab/AwesomeLearningAPR .

Variable-based Fault Localization via Enhanced Decision Tree

Fault localization, aiming at localizing the root cause of the bug under repair, has been a longstanding research topic. Although many approaches have been proposed in past decades, most of the existing studies work at coarse-grained statement or method levels with very limited insights about how to repair the bug ( granularity problem ), but few studies target the finer-grained fault localization. In this article, we target the granularity problem and propose a novel finer-grained variable-level fault localization technique. Specifically, the basic idea of our approach is that fault-relevant variables may exhibit different values in failed and passed test runs, and variables that have higher discrimination ability have a larger possibility to be the root causes of the failure. Based on this, we propose a program-dependency-enhanced decision tree model to boost the identification of fault-relevant variables via discriminating failed and passed test cases based on the variable values. To evaluate the effectiveness of our approach, we have implemented it in a tool called VarDT and conducted an extensive study over the Defects4J benchmark. The results show that VarDT outperforms the state-of-the-art fault localization approaches with at least 268.4% improvement in terms of bugs located at Top-1, and the average improvement is 351.3%. Besides, to investigate whether our finer-grained fault localization result can further improve the effectiveness of downstream APR techniques, we have adapted VarDT to the application of patch filtering, where we use the variables located by VarDT to filter incorrect patches. The results denote that VarDT outperforms the state-of-the-art PATCH-SIM and BATS by filtering 14.8% and 181.8% more incorrect patches, respectively, demonstrating the effectiveness of our approach. It also provides a new way of thinking for improving automatic program repair techniques.

APR4Vul: an empirical study of automatic program repair techniques on real-world Java vulnerabilities

Security vulnerability fixes could be a promising research avenue for Automated Program Repair (APR) techniques. In recent years, APR tools have been thoroughly developed for fixing generic bugs. However, the area is still relatively unexplored when it comes to fixing security bugs or vulnerabilities. In this paper, we evaluate nine state-of-the-art APR tools and one vulnerability-specific repair tool. In particular, we investigate their ability to generate patches for 79 real-world Java vulnerabilities in the Vul4J dataset, as well as the level of trustworthiness of these patches. We evaluate the tools with respect to their ability to generate security patches that are (i) testable, (ii) having the positive effect of closing the vulnerability, and (iii) not having side effects from a functional point of view. Our results show that the evaluated APR tools were able to generate testable patches for around 20% of the considered vulnerabilities. On average, nearly 73% of the testable patches indeed eliminate the vulnerabilities, but only 44% of them could actually fix security bugs while maintaining the functionalities. To understand the root cause of this phenomenon, we conduct a detailed comparative study of the general bug fix patterns in Defect4J and the vulnerability fix patterns in ExtraVul (which we extend from Vul4J). Our investigation shows that, although security patches are short in terms of lines of code, they contain unique characteristics in their fix patterns compared to general bugs. For example, many security fixes require adding method calls. These method calls contain specific input validation-related keywords, such as encode, normalize, and trim. In this regard, our study suggests that additional repair patterns should be implemented for existing APR tools to fix more types of security vulnerabilities.

Multi-View Feature Fusion Model for Software Bug Repair Pattern Prediction

Many search-based Automatic Program Repair (APR) techniques employ a set of repair patterns to generate candidate patches. Regarding repair pattern selection, existing search-based APR techniques either randomly select a repair pattern from the repair pattern set to apply or prioritize all repair patterns based on the bug's context information. In this paper, we introduce PatternNet, a multi-view feature fusion model capable of predicting the repair pattern for a reported software bug. To accomplish this task, PatternNet first extracts multi-view features from the pair of buggy code and bug report using different models. Specifically, a transformer-based model (i.e., UniXcoder) is utilized to obtain the bimodal feature representation of the buggy code and bug report. Additionally, an Abstract Syntax Tree (AST)-based neural model (i.e., ASTNN) is employed to learn the feature representation of the buggy code. Second, a co-attention mechanism is adopted to capture the dependencies between the statement trees in the AST of the buggy code and the textual tokens of the reported bug, resulting in co-attentive features between statement trees and reported bug's textual tokens. Finally, these multi-view features are combined into a unified representation using a feature fusion network. We quantitatively demonstrate the effectiveness of PatternNet and the feature fusion network for predicting software bug repair patterns.

Mining on Students’ Execution Logs and Repairing Compilation Errors Based on Deep Learning

Automatic program repair techniques based on deep neural networks have attracted widespread attention from researchers due to the high degree of automation and generality. However, there is a scarcity of high-quality labeled datasets available for training program repair models. This study proposes a method of mining reasonable program repair examples from student program execution logs. Additionally, we introduce the Rookie Simulator (RS), which simulates the error patterns commonly made by novice programmers and generates a large number of program repair sample pairs. To address the issue of low repair rates for infrequent and complex error patterns in compilation errors, the study proposes the attention-enhanced capsule network for program repair (ACNPR), a program repair model that integrates compiler feedback information and utilizes capsule networks to capture complex semantic features. Experimental evaluations were conducted using publicly available datasets, including the DeepFix, TEGCER, and a real course dataset named SUES-COJ mined in this study. The results indicate that our method consistently outperforms current state-of-the-art models in terms of full repair rates.

Seeing the Whole Elephant: Systematically Understanding and Uncovering Evaluation Biases in Automated Program Repair

Evaluation is the foundation of automated program repair (APR), as it provides empirical evidence on strengths and weaknesses of APR techniques. However, the reliability of such evaluation is often threatened by various introduced biases. Consequently, bias exploration, which uncovers biases in the APR evaluation, has become a pivotal activity and performed since the early years when pioneer APR techniques were proposed. Unfortunately, there is still no methodology to support a systematic comprehension and discovery of evaluation biases in APR, which impedes the mitigation of such biases and threatens the evaluation of APR techniques. In this work, we propose to systematically understand existing evaluation biases by rigorously conducting the first systematic literature review on existing known biases and systematically uncover new biases by building a taxonomy that categorizes evaluation biases. As a result, we identify 17 investigated biases and uncover a new bias in the usage of patch validation strategies. To validate this new bias, we devise and implement an executable framework APRConfig , based on which we evaluate three typical patch validation strategies with four representative heuristic-based and constraint-based APR techniques on three bug datasets. Overall, this article distills 13 findings for bias understanding, discovery, and validation. The systematic exploration we performed and the open source executable framework we proposed in this article provide new insights as well as an infrastructure for future exploration and mitigation of biases in APR evaluation.

Patching Locking Bugs Statically with Crayons

The Linux Kernel is a world-class operating system controlling most of our computing infrastructure: mobile devices, Internet routers and services, and most of the supercomputers. Linux is also an example of low-level software with no comprehensive regression test suite (for good reasons). The kernel’s tremendous societal importance imposes strict stability and correctness requirements. These properties make Linux a challenging and relevant target for static automated program repair (APR). Over the past decade, a significant progress has been made in dynamic APR. However, dynamic APR techniques do not translate naturally to systems without tests. We present a static APR technique addressing sequential locking API misuse bugs in the Linux Kernel. We attack the key challenge of static APR, namely, the lack of detailed program specification, by combining static analysis with machine learning to complement the information presented by the static analyzer. In experiments on historical real-world bugs in the kernel, we were able to automatically re-produce or propose equivalent patches in 85% of the human-made patches, and automatically rank them among the top three candidates for 64% of the cases and among the top five for 74%.

Effective Isolation of Fault-Correlated Variables via Statistical and Mutation Analysis

It is a widely-adopted strategy for developers to monitor the values of program variables when debugging in practice. In particular, developers often set breakpoints at specific locations or execute the program step by step in the debugging mode to inspect if abnormal values or status will be observed for concerned variables. Such a practical debugging strategy can facilitate developers in understanding and localizing the target fault. This study aims to identify suspicious program variables of a given fault (i.e., denoted as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">fault-correlated variables</i> ) automatically, thus facilitating the debugging activities for developers. To the best of our knowledge, this is the finest granularity in fault localization (FL) so far, which can address the limitations of being coarse-grained as faced by existing FL techniques. However, isolating fault-correlated variables precisely is challenging since there are usually substantially different variables used or defined in a program, and plenty of them are in the same basic block which cannot be well discriminated from each other since they will be either executed or not against the given test suite. To address such challenges, this study presents IsoVar, a two-phase model to isolate fault-correlated variables. Specifically, IsoVar first performs statistical analysis based on <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">variable execution matrices</i> , which is a novel concept proposed in this study, to identify a set of suspicious variables. It then observes the impacts of those variables on the program dynamically after applying subtle mutations at the bytecode level, to further isolate fault-correlated variables. Extensive experiments on Defects4J and Bears demonstrate that IsoVar can outperform state-of-the-art techniques significantly ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$13.0\%$</tex-math></inline-formula> for MAP and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$19.3\%$</tex-math></inline-formula> for MRR). More importantly, we incorporated IsoVar into 11 existing FL techniques as well as 14 automated program repair techniques, and found that IsoVar can significantly boost their performance.

SSDTutor: A feedback-driven intelligent tutoring system for secure software development

Application Programming Interfaces (APIs) in cryptography typically impose concealed usage constraints. The violations of these usage constraints can lead to software crashes or security vulnerabilities. Several professional tools can detect these constraints (API misuses) in cryptography; however, in the educational programs, the focus has been less on helping students implement an application without cryptographic API misuses that are caused by either a lack of cryptographic knowledge or programming mistakes. To address the problem, we present an intelligent tutoring approach SSDTutor for educating Secure Software Development. Our tutoring approach helps students or developers repair cryptographic API misuse defects by leveraging an automated program repair technique based on the usage patterns of cryptographic APIs. We studied the best practices of cryptographic implementations and encoded eight cryptographic API usage patterns. For quality feedback, we leverage a clone detection technique to recommend related feedback for helping students understand why their programs are incorrect, rather than blindly accepting repairs. We evaluated SSDTutor on 456 open source subject projects implemented with cryptographic APIs. SSDTutor successfully detected 1,553 out of 1,573 misuse defects with 98.9% accuracy and repaired 1,551 out of 1,573 misuse defects with 99.3% accuracy. In a user study involving 22 students, the participants reported that interactive SSDTutor's feedback recommendation could be valuable for novice students to learn about the correct usages of cryptography APIs.

Let’s Talk With Developers, Not About Developers: A Review of Automatic Program Repair Research

Automatic program repair (APR) offers significant potential for automating some coding tasks. Using APR could reduce the high costs historically associated with fixing code faults and deliver significant benefits to software engineering. Adopting APR could also have profound implications for software developers’ daily activities, transforming their work practices. To realise the benefits of APR it is vital that we consider how developers feel about APR and the impact APR may have on developers’ work. Developing APR tools without consideration of the developer is likely to undermine the success of APR deployment. In this paper, we critically review how developers are considered in APR research by analysing how human factors are treated in 260 studies from Monperrus’s Living Review of APR. Over half of the 260 studies in our review were motivated by a problem faced by developers (e.g., the difficulty associated with fixing faults). Despite these human-oriented motivations, fewer than 7% of the 260 studies included a human study. We looked in detail at these human studies and found their quality mixed (for example, one human study was based on input from only one developer). Our results suggest that software developers are often talked <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">about</i> in APR studies, but are rarely talked <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">with</i> . A more comprehensive and reliable understanding of developer human factors in relation to APR is needed. Without this understanding, it will be difficult to develop APR tools and techniques which integrate effectively into developers’ workflows. We recommend a future research agenda to advance the study of human factors in APR.

Patch It If You Can: Increasing the Efficiency of Patch Generation Using Context

Although program repair is a tremendous aspect of a software system, it can be extremely challenging. An Automated Program Repair (APR) technique has been proposed to solve this problem. Among them, template-based APR shows good performance. One of the key properties of the template-based APR technique for practical use is its efficiency. However, because the existing techniques mainly focus on performance improvement, they do not sufficiently consider the efficiency. In this study, we propose EffiGenC, which efficiently explores the patch ingredient search space to improve the overall efficiency of the template-based APR. EffiGenC defines the context using the concept of extended reaching definition from compiler theory. EffiGenC constructs the search space by collecting the ingredient required for patching in the context. We evaluated EffiGenC on the Defects4j benchmark. EffiGenC decreases the number of candidate patches from 27% to 86% compared to existing techniques. EffiGenC also correctly/plausibly fixes 47/72 bugs. For Future work, we will solve the search space problem that exists in multiline bugs using context.

Automatic Repair of Java Programs Weighted Fusion Similarity via Genetic Programming

Recently, automated program repair techniques have been proven to be useful in the process of software development. However, how to reduce the large search space and the random of ingredient selection is still a challenging problem. In this paper, we propose a repair approach for buggy program based on weighted fusion similarity and genetic programming. Firstly, the list of modification points is generated by selecting modification points from the suspicious statements. Secondly, the buggy repair ingredient is selected according to the value of the weighted fusion similarity, and the repair ingredient is applied to the corresponding modification points according to the selected operator. Finally, we use the test case execution information to prioritize the test cases to improve individual verification efficiency. We have implemented our approach as a tool called WSGRepair. We evaluate WSGRepair in Defects4J and compare with other program repair techniques. Experimental results show that our approach improve the success rate of buggy program repair by 28.6%, 64%, 29%, 64% and 112% compared with the GenProg, CapGen, SimFix, jKali and jMutRepair.

A comprehensive study of code-removal patches in automated program repair

Automatic Program Repair (APR) techniques can promisingly help reduce the cost of debugging. Many relevant APR techniques follow the generate-and-validate approach, that is, the faulty program is iteratively modified with different change operators and then validated with a test suite until a plausible patch is generated. In particular, Kali is a generate-and-validate technique developed to investigate the possibility of generating plausible patches by only removing code. Former studies show that indeed Kali successfully addressed several faults. This paper addresses the single and particular case of code-removal patches in automated program repair. We investigate the reasons and the scenarios that make their creation possible, and the relationship with patches implemented by developers. Our study reveals that code-removal patches are often insufficient to fix bugs, and proposes a comprehensive taxonomy of code-removal patches that provides evidence of the problems that may affect test suites, opening new opportunities for researchers in the field of automatic program repair.

SituRepair: Incorporating machine-learning fault class prediction to inform situational multiple fault automatic program repair

Automatic program repair (APR) is an active research area, in which the problem is to discover a solution for a given faulty program that fixes the fault. As yet, a myriad of APR techniques has been proposed in the literature even though the research should go a long way toward establishing APR techniques that are industrially-viable. A current issue is to deal with multiple-fault (MF) programs. Although they are prevalent in real-world settings, automatic repair of MF programs has received less attention in the literature. In this paper, we propose SituRepair, a new APR technique which is developed to fix MF programs; that is, it is intended to discover a solution that fixes multiple faults at a time for a given faulty program. The significance of SituRepair for Critical Infrastructure Protection (CIP) is to provide extensive empirical evidence that APR is viable in CIP software systems. First, we compute 50 static features for each faulty program in a training benchmark where the type and situation of the fault(s) are known. We have considered 151 types of different faults, which have been manually elicited from the training benchmark. The fault types include various single-fault (SF) and MF cases. Then, we use the faulty programs’ static data to build a machine-learning model. Next, for a new faulty program, the model receives as input the values of its static features and predicts the types and situations of the fault(s). Finally, we apply various modifications to the faulty program, according to the types and situations of the fault(s), so that every fault is fixed. SituRepair employs a program's test suite to evaluate the modified variants, as test suites are often readily-available. To investigate the effectiveness of SituRepair, we have conducted extensive experiments on 25,619 C programs within Code4Bench, including 16,274 SF and 9345 MF programs. In the experiments, SituRepair could successfully repair 17,195 programs of which 13,347 cases are SF and 3848 cases are MF programs. We conclude that SituRepair seems to have the potential to be used for multiple fault APR. In addition, SituRepair could repair each program in 9.24 min on average, which is of practical importance. The impact of this work on researchers is to incentivize them to apply further static code features and sophisticated machine-learning models. For practitioners, it can facilitate the debugging tasks yielding higher quality software systems for CIP settings.

Review4Repair: Code review aided automatic program repairing

Learning-based automatic program repair techniques are showing promise to provide quality fix suggestions for detected bugs in the source code of the software. These tools mostly exploit historical data of buggy and fixed code changes and are heavily dependent on bug localizers while applying to a new piece of code. With the increasing popularity of code review, dependency on bug localizers can be reduced. Besides, the code review-based bug localization is more trustworthy since reviewers’ expertise and experience are reflected in these suggestions.

Improving Search-Based Automatic Program Repair With Neural Machine Translation

The challenge of automatically repairing bugs in programs to reduce debugging expenses and increase program quality is known as automated program repair. To overcome this issue, test-suite-based repair techniques use a specified test suite as an oracle and alter the input faulty program to pass the full test suite. GenProg is a well-known example of this kind of repair, in which genetic programming is used to reorder the statements already present in the faulty program. However, recent practical experiments suggest that GenProg’s performance, notably for Java, is not sufficient. Improved program dependability necessitates the use of automatic program repair techniques. Template-based program repair techniques have recently been combined with search-based techniques to solve program issues automatically. Although intriguing, it has two fundamental drawbacks: Its search space often lacks the correct solution, and the technique disregards program expertise, such as precise code language. Compared with the template-based program repair approach, existing neural-machine-translation-based approaches are not limited by these constraints due to their ability to learn and generate new solutions. We propose an approach that combines a search-based automatic program repair technique with a neural-machine-translation-based approach. More specifically, we use both redundancy assumption and sequence-to-sequence learning of correct patches as the source for potential fix statements that feed into a multiobjective evolutionary search algorithm to find test-suite-adequate patches. In this work, a novel framework called ARJANMT is introduced for automatically repairing Java programs. Two sets of controlled experiments are conducted on 410 bugs from two benchmarks to investigate the repairability and correctness of our proposed framework. A comparison between state-of-the-art automatic program repair frameworks is made. The experimental results indicate that combining those two types of repair techniques (search-based and neural-machine-translation-based) produces better results or fixes bugs that they previously were unable to fix individually.

SOSRepair: Expressive Semantic Search for Real-World Program Repair

Automated program repair holds the potential to significantly reduce software maintenance effort and cost. However, recent studies have shown that it often produces low-quality patches that repair some but break other functionality. We hypothesize that producing patches by replacing likely faulty regions of code with semantically-similar code fragments, and doing so at a higher level of granularity than prior approaches can better capture abstraction and the intended specification, and can improve repair quality. We create SOSRepair, an automated program repair technique that uses semantic code search to replace candidate buggy code regions with behaviorally-similar (but not identical) code written by humans. SOSRepair is the first such technique to scale to real-world defects in real-world systems. On a subset of the ManyBugs benchmark of such defects, SOSRepair produces patches for 22 (34%) of the 65 defects, including 3, 5, and 6 defects for which previous state-of-the-art techniques Angelix, Prophet, and GenProg do not, respectively. On these 22 defects, SOSRepair produces more patches (9, 41%) that pass all independent tests than the prior techniques. We demonstrate a relationship between patch granularity and the ability to produce patches that pass all independent tests. We then show that fault localization precision is a key factor in SOSRepair's success. Manually improving fault localization allows SOSRepair to patch 23 (35%) defects, of which 16 (70%) pass all independent tests. We conclude that (1) higher-granularity, semantic-based patches can improve patch quality, (2) semantic search is promising for producing high-quality real-world defect repairs, (3) research in fault localization can significantly improve the quality of program repair techniques, and (4) semi-automated approaches in which developers suggest fix locations may produce high-quality patches.

E-APR: Mapping the effectiveness of automated program repair techniques

E-APR: Mapping the effectiveness of automated program repair techniques

CGenProg: Adaptation of cartesian genetic programming with migration and opposite guesses for automatic repair of software regression faults

In the last decade, the research community has been actively working to develop the techniques that can automatically find a solution to a software fault, namely, automatic program repair (APR). As of today, a multitude of APR techniques has been proposed. The techniques could have effectively repaired a wide variety of fault classes. However, the development of effective APR techniques for software regression faults, which are prevalently occurred in the maintenance stage of the software lifecycle, have received little attention. By incorporating specific knowledge in the domain of software regression faults, we have developed a novel technique for automatic repair of software regression faults in Java programs, which we call CGenProg. To achieve this, we have extensively adapted and modified the original cartesian genetic programming (CGP), biogeography-based optimization (BBO), and opposition-based learning (OBL). The modified CGP serves us as the core evolutionary process while the modified BBO and OBL act as crossover and mutation, respectively. The significance of CGenProg is that it contributes to the solution of a practical problem faced by developers in the maintenance stage of the software lifecycle. For expert and intelligent systems, it extends what is known about the application of optimization algorithms in the context of APR. Further, it demonstrates a novel use of CGP, BBO, and OBL for automatic repair of software regression faults. To evaluate CGenProg, we have developed a prototype tool using the Java language. Then, we conducted experiments on several programs in Code4Bench where each program is released with multiple consecutive versions comprising software regression faults. In the experiments, CGenProg could repair 17 out of 30 faulty programs. We conclude that CGenProg proves relevant and effective for repairing software regression faults. The impact of this study is to incentivize researchers for further exploitation and adaptation of the wealth of existing metaheuristics to develop effective APR techniques for different fault classes.